General Disclaimer 


One or more of the Following Statements may affect this Document 


• This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 


• This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 


• This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 


• This document is paginated as submitted by the original source. 


• Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 


Produced by the NASA Center for Aerospace Information (CASI) 




AgRISTARS 

“Made available NASA sponsorship 

in the interesi 'Jf ■ and wide dis- 
semination 0 i t r. ^-resources Survey 
Program informa-'^'n and without liability 
for any use made thereof,’’ 

g Foreign Commodity 
‘ Production Forecasting^ 


£82' 1023 

FC-LO-00472 
JSC-16752 ^ / 

OCT 2 4 

A Joint Program for 
Agriculture and 
Resources Inventory 
Surveys Through 
Aerospace 
Remote Sensing 

August 1980 



EVALUATION OF THE PROCEDURE FOR SEPARATING 
BARLEY FROM OTHER SPRING SMALL GRAINS 



E. R. Magness 


10230) EVALUiiliuO Ui<’ tiLl: t-nOCiLUKi iOH Nu,i-^3u03 

I5BPA2A2ING BABLLjf i-BCK CTh3;& OKALL 

GliAXIvS (xockiiuoa nugxnt.'«L iiiy ai.d Matidceiiient) 

77 p HC AOS/MF A0l OSCl 02C Uncial 

G3/4J 00230 


-^1 


It 



Lockheed Engineering and Management Services Company, Inc. 
1830 Ni-ISA Road 1, Houston, Texas 77053 



!W>S/\ 






(MO fTAlIt 9 


mill 


P 


Lyndon B. Johnson Space Center 

Houston, Texas 77058 



jiii 






Mi. -Ji-.....- ■ ...'Ia*..*.. ,J 



1, Rtpori No, 2. Govornmani Accaiiion No. . 

FC-LO-00472; JSC-16752 

3. Rocipiani'f Caulog No. 

4, Tllla and Subntlt 

Evaluation of the Procedure for Separating Barley from Other 
Spring Small Grains 

5. fltpon Oatt 1 

August 1980 

6. Farforming Organuatlon Coda 

7, Authorfi) 

E- r*. Magness 

8. Farforming Organiia^-mb^ No. 

LEMSCO-14598 

10. Work Unit No, 

9. ^forming Organljaiion Namt and Addrt« 

Lockheed Engineering and Management Services Company, Inc. 
1830 NASA Road 1 
Houston, Texas 77058 

11. ConuKt or GranrJVo. 

NAS 9-15800 ( 5j 

13, Typa of Raport and' Pariod Covarad 

Technical Report 

12, Spomoring Agoncy Nam* and Addrtu 

National Aeronautics and Space Administration / 1 J 

Lyndon B. Johnson Space Center LJ/ 

Houston, Texas 77058 (Technical Monitor: R. 0. Hill/SF4) 

14, Sponioring AgirKy Coda 

IS. Supplamantary Nott« 

16. Abitrici 


The major objectives of the investigation documented in this report are to evaluate the 
procedure for separating barley from the other small grains and to determine if the 
assumptions upon which the procedure is based hold true. Personnel who developed the 
procedure exercised the procedure by processing sample segments in the U.S. Northern 
Great Plains, The results of this processing were then compared to ground truth to obtain 
accuracy measurements. Results of testing the procedure indicate no difficulty in exercising 
the procedure correctly; however* there were differences in the labeling accuracies obtained 
by the evaluators from those obtained by the TY analysts. Reasons for some differences and 
a discussion of the procedural strengths and weaknesses are offered in the conclusions. 


17, Kay Words (Suggaitad by Author(t|) 

Barley separation procedure 
Labeling accuracy 
Ground truth 

18. Distribution Statamant 

19, Spcurity Qaaif. (of tbii raporil 

Unclass.if ied 

20, Sacurity Ctauif, (of this pagal 

Unclassified 

21. No, of Pagas 

76 

22. Ptie*' 


*For ulc by <bt National Tachnical Information Sarvico, Sprjiigfiald. Virginia 22161 
iSC Form 14J4 (Hw No» 791 WASA — JSC 












m 



FC-LO-00472 

JSC-16752 


EVALUATION OF THE PROCEDURE FOR SEPARATING BARLEY FROM 
OTHER SPRING SMALL GRAINS 


Job Order 74-402 


This report describes Classification activities of the Foreign 
Commodity Production For^'casting project of the AgRISTARS program. 


PREPARED BY 
E. R. Magness 


APPROVED BY 






"L. M. Flores, Supervisor 
Design Integration Section 


LOCKHEED ENGINEERING AND MANAGEMENT SERVICES COMPANY, INC. 
Under Contract NAS 9-15800 
For 

Earth Observations Division 

Space and Life Sciences Directorate 

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 
LYNDON B. JOHNSON SPACE CENTER 
HOUSTON. TEXAS 

August 1980 


LEMSCO-14598 


PI^ECEDING PAGE BLAWK NOT FILMED 


PREFACE 

The main objective of this study was the evaluation of the procedure which 
separates barley from the other spring grains and which was developed for 
processing of the Transition Year segments. For this evaluation, a decision 
was made specifying that the procedure developers themselves would sxercise 
the procedure in order not to confound procedural problems with implementation 
problems. Thus, it became necessary for the evaluation to proceed by first 
labeling the spring small grains. 

The accuracy of the spring small-grain labels obtained in the labeling effort 
were, on the average, somewhat better than that in the Transition Year opera- 
tions. The more accurate labels are thought to be due to improvements in the 
procedure for identification and labeling. Specifically, the departures from 
the previous procedure included a regionalization of the labeling process, the 
use of trend analysis, and the removal of time constraints from the actual 
processing. However,’ It is not within the scope of this report to include 
assessments of the labeling accuracies obtained here versus those obtained in 
the Transition Year operations. Therefore, no specific treatment of these 
differences will be discussed. Some comments are in order and are included in 
the conclusion. 


V 
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1. INTRODUCTION 


1.1 BACKGROUND 

An objective of the Large Area Crop Inventory Experiment (LACIE) was to deter- 
mine wheat acreage (ref. 1); however, throughout LACIE Phases I, II, and III, 
the analysts in the Classification and Mensuration Subsystem (CAMS) generated 
proportion estimates of all the spring small -grain and winter small -grain 
crops that they could identify within the 5- by 6-nautlcal-mne segments. In 
the regions where spring small -grain crops are grown, it was not possible for 
the LACIE analysts to separate the spring wheat from the other small grains. 
Therefore, It became necessary to mathematically proportion the other spring 
small grains (and flax, a confusion crop) and the spring wheat based on 
historical or other information (ref. 2). Thus, the inability to estimate 
wheat acreage directly remained an unresolved issue during LACIE, 

In an initial developmental eff>rt prior to the processing of LACIE Phase III 
data, a procedure was developed for estimating spring wheat. Guidelines based 
on LACIE Phase II data were used. This initial procedure was tested by making 
estimates cn 18 North Dakota blind sites using LACIE Phase III data. A 
description of this procedure and the results obtained from its use are in 
reference 3. The procedure was only marginally successful, but the report 
showed that some of the procedural guidelines used were Indeed valid. In par- 
ticular, evaluation of the spectral aids in the form of green number versus 
brightness scatter plots of the grain categories [Kauth greenness and bright- 
ness (ref. 4)] showed potential for separating the spring grains provided that 
an acquisition at the correct growth stage was used. 

In early 1978, the author of this report and others used data from 46 LACIE 
Phase III blind-site segments in the four U.S. Northern Great Plains (USNGP) 
states to develop a procedure which could be utilized to separate barley from 
the other spring small grains. The procedure relied on the observation that 
barley tends to become brighter than wheat during and after the soft-dough 
stage. This finding was later confirmed by W. A. Mali la (ref. 5). 
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The current procedure, which Is the subject of this report, was developed by 
using a combination of spatial and color details observed in the production 
film converter (PFC) products and spectral aids. Large-scale testing of the 
procedure was subsequently undertaken using the LAC IE Transition Year (TY) 
Landsat data. During TY, three proportion estimates were to be produced from 
the spring-grain analysis. They were a “B" for barley estimate; an "S" for 
spring-wheat, plus oats, plus flax estimate; and a "W“ for winter-wheat esti- 
mate, when applicable. Ratios would still be applied using a historical 
econometric model (ref. 6) to the S estimate in order to arrive at the final 
spring-wheat estimate. When S and B estimates were not possible through lack 
of the necessary Landsat data, the S and B estimates woulu be derived from the 
current spring-grain estimate using a regression model derived from other 
segments in the area (ref. 7). 

1.2 OBJECTIVES 

The objective of this study is to evaluate that portion of the direct wheat 
procedure used for labeling Landsat data as barley and other spring-grain 
signatures. The procedure, developed for use during TY, is documented in 
reference 8. The specific objectives of this study are: 

a. To determine if the assumptions about the behavior of the relevant crops 
(upon which the procedure is based) hold true with the TY data [This 
includes studying the extendability of the procedure from the years upon 
which the development was based (crop years 1976 and 1977) to another year 
(crop year 1978).] 

b. To measure how well key acquisitions are selected for separation and how 
well the decision boundaries are placed on the scatter plots (This 
includes studying the sensitivity of the results generated using this 
procedure to the errors in selection of the key acquisition.) 

c. To evaluate the applicability of the procedure when exercised in larger 
geographic areas of varying soils, varying meteorological conditions, and 
varying cropping practices 

d. To provide recommendations for further improvements of the procedure 
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As stated previously, the major objectives of this Investigation were to 
evaluate the procedure for separating barley from the other small grains and 
to determine If the assumptions upon which the procedure Is based hold true. 

In order to perform this evaluation using TY operational results, it would 
have been necessary to mount a separate but coordinated effort to assess TY 
operational errors. This type of assessment Is both costly and subjective. 

The results of such an Investigation would also be confounded with potential 
Implementation problems. Therefore, TY operational results were not used. 
Instead, procedures development personnel themselves assessed the success of 
the procedure by processing a sample of segments. The results of this 
processing were then compared to ground truth to obtain accuracy measurements. 
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2. DATA SET 


2.1 SELECnON OF SEGMENTS FOR EVALUATION 

Blind sites are a group of segments representing a random sampling of the seg- 
ments for spring small grains in the U.S. Great Plains (USGP) for which wall* 
to-wal’l ground-truth inventories are obtainisd. Blind sites were used as the 
ground-truth source during evaluation of the procedure. 

The following criteria determined which segments were to be used in this 
study: 

a. The TY blind sites with ground truth were in one of the four USNGP 

states: North Dakota, South Dakota, Minnesota, or Montana. 

b. Segments were required to have sufficient cloud-free Landsat acquisitions 
with no technical problems, allowing accurate identification and estima- 
tion of spring small grains as specified in reference 8, f’^ragraph 4.2. 

2.2 GROUND-TRUTH DERIVATION 

Ground truth for each segment evaluated was derived using the 1:24 000-scale 
annotated blind-site color infrared photography provided by the Agricultural 
Stabilization Conservation Service (ASCS) of the U.S. Department of 
Agriculture (USDA). Each of the 209 dots labeled in each segment was compared 
to the corresponding field in the annotated ASCS photograph. The reference 
Landsat acquisition used in making the comparison was the one used for barley 
labeling when the segment was analyzed. GroL;nd-truth labels for the 209 dots 
were determined by experienced analysts comparing directly the ground-truth- 
annotated photography with the Landsat imagery. Tables B-l(a) through B-l(d) 
in appendix B provide a comparison of the 209-dot random estimates with the 
209-dot ground truth percentages. 
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2.3 DATA AVAILABLE FOR EACH SEGMENT USED IN ANALYSIS 


The USNGP states are presented in figure 2-l(a); the geographic locations of 
the blind- site segments used in this study are plotted on the reference map in 
figure 2-1 (b). 

In appendix A, table A-1 is a tabulation of the types of data available and 
used during the original analysis of each of the 45 segments selected* Each 
segment is believed to have at least sufficient acquisitions to identify 
spring grains as required in reference 8. Additional data such as Landsat 
full frames (when indicated), historical county-acreage percentages, refer- 
ences to blind-site signatures and percentages of previous years, and CAMS 
estimates from previous years were used as aids in spring-grain analysis as a 
matter of analyst preference. In addition to the aforementioned data for each 
segment, a historical crop calendar for each U.S. Crop Repor\,ing District 
(CRD) and adjustments provided by the National Oceanic and Atmospheric 
Administration (NOAA) to reflect real-time segment spring-wheat conditions 
were available at the tiine of analysis for all segments. Significant impact 
made by each of the above types of reference data will be mentioned where 
appropriate throughout the report. 

The bulk of detail in table A-1 of appendix A is provided to indicate exactly 
what data were available and used in analyzing each segment. The "late- 
heading-to-ripe" acquisition used for barley separation and labeling in each 
segment is designated in table A-1 by footnote. 

In the “Remark" column of table A-1, the information indicates which segments 
meet the criteria of Code 33, a code indicating the absence of an acquisition 
showing the barley emergent signature as defined in section 5.3. The remark 
column also indicates those segments which were analyzed for the first time 
during TY or which had been relocated since LACIE Phase III processing. It 
should be noted here that historical county grain-acreage percentages were 
available for analyst use in North Dakota only on new or relocated segments. 
These data were available in other states for all segments. 
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Figure 2-l(a).- The USNGP states. 







3. THE PROCEDURE 


3.1 ASSUMPTIONS 

Listed below are the principles or basic assumptions upon which the procedure 

for the separation of TY spring small grains and barley is based. 

♦ 

a. Barley matures earlier than other spring grains. 

b. Barley becomes brighter and less green within a definite growth-stage 
range and can be identified (separated from other small grains) using 
color changes which occur at this stage in the Landsat imagery products 
and in the green number versus brightness scatter plots of the 209 dots. 

3.2 APPLICATIO N 

The' procedure requires that the analysts identify all spring small grains and 
determine whether the correct acquisition is present to allow the separation 
of barley. For those segments not having the critical barley acquisition, the 
analyst labels the segment for total spring small grains only end does not 
attempt to label barley. The following steps are to be completed for those 
segments that do have an acquisition in the barley separation window; these 
steps are taken after the spring grains have been identified in accordance 
with reference 8. 

a. Identify the acquisition in the barley separation window. (This is the 
acquisition in which fields of the least mature spring grains have reached 
the late-heading stage but in which none have been harvested.) 

b. Label the more advanced (brighter and yellower) signatures in the PFC pro- 
ducts as definite barley (B); the less advanced signatures (less bright 
and more red) as other spring small grains (S); the mixed or uncertain 
spring small -grain signatures (V or Q). Label all 209 dots in the dot 
grid of the segment identifying all the spring small grains with one of 
the four alphabetic characters: B, S, V, or Q. 

c. Generate a green number versus brightness scatter plot (fig. 3-1) of the 
spring small -grain dots. In those cases in which barley is separable, the 
barley dots assume a scattered distribution apart from the cluster of the 
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Green number 



0 * • • • • • • • 

20 30 40 50 60 70 80 90 

Brightness 

Legend: 

B Barley dot correctly Identified 0 Oats dot correctly identified 

Barley dot labeled nongrain (omission) S Spring-wheat (or durum) dot correctly identified 

B. Barley dot labeled S (confusion) © Spring-wheat dot labeled nongrain (omission) 

C Nongrain labeled S (commission) S, Spring-wheat dot labeled B (confusion) 

t Nongrain labeled B (commission) — Analysts decision boundary 

F Flax dot correctly identified --- Optimal decision boundary 

© Flax dot labeled nongrain (omission) 
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other grains. Specificany, the barley dots will be generally below and 
to the right of the main spring-grains cluster. A comparison of scatter 
plots at 18-day intervals will illustrate to the reader the general drift 
of the barley dots as they separate from the other small grains. Examples 
are contained in reference 8 which documents the procedure in detail. 

d. Decide the final labels for those dots of questionable identity by using 
the dot distribution in the scatter plot and the color details in the 
imagery. If some dots are very difficult to label, they should be propor- 
tioned as B or S dots in the same ratio as the B and S dots already 
identified. 
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4. APPROACH 


4.1 INTRODUCTION 

Two analysts were designated to evaluate the procedure. These analysts were 
both highly experienced and heavily involved in the development phase of the 
procedure. The analysts themselves implemented the procedure on test sites 
having available ground truth. The results were then evaluated using the 
ground truth. The barley procedure was impiemented by executing the whole 
labeling procedure for spring small grains as described in reference 8. 

First, the spring small-grain dots were identified and labeled; then, the 
barley dots were separated from the other spring small -grain dots. Procedures 
similar to those used in other LACIE accuracy assessment reports were used to 
evaluate the results. These procedures are documented in reference 9. 

4.2 LABEL TERMINOLOGY USED 

For purposes of this study, the following symbols will apply when referring to 
the spring small grains: 

SW = spring wheat 

51 = oats 

B = barley 

F = flax 

S = spring wheat + oats + flax 
SG = S B (spring small grain or spring grain) 

4.3 METHODOLOGY 

The labels obtained by consensus opinion of the two analysts were recorded on 
dot-label forms in use by LACIE analysts. These labels and the corresponding 
ground-truth labels, as determined by the analysts using the ASCS-annotated 
1:24 000-scale aerial photography, were subsequently recorded on a form 
designed to allow dot-by-dot comparisons (ref. 9, fig. 1). To estimate the 
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degree of success of the labeling experiment, the following detenni nations 
were made from the analyst labels and the ground-truth labels for the 
segments; 

a. Random estimates of the proportion of spring small grains in the segments 
were computed from the labels of the 209 grid dots and compared with the 
wall-to-wall ground-truth random-dot proportion estimations. 

b. Labeling accuracy estimates were obtained by direct comparison of the 
labels of dots determined by analysts to corresponding labels derived 
using ground-truth infonnation. Ground-truth labels were determined by 
direct comparison of an annotated ASCS photograph to the appropriate 
Landsat acquisition (base date). The accuracy calculations included omis- 
sion and commission errors of the categories of interest. The categories 
of interest include spring small grains (SG) and nonspring small grains 
(N). The spring small grains were further categorized into spring small 
grains less barley [spring wheat + oats + flax (S)] and barley (B). 

c. Error causes were obtained by characterizing the errors in labeling using 
procedures previously developed and used in LACIE (ref. 9). These proce- 
dures consist of a subjective attempt to characterize labeling errors. 

The analysis was done for each labeled dot and the results were sumnarized 
at the state level. 

d. Scatter plots of the key acquisitions with ground-truth labels were gener- 
ated to aid in the assessment of errors in the barley separation process. 
Specifically, they were used to determine if the barley behaved as 
expected and if it were separable on the expected acquisition or on an 
alternate acquisition. Conversely, if the barley exhibited the expected 
behavior, the scatter plots were used to determine if the decision 
boundary was misplaced. 


5. RESULTS 


5.1 SPRING-GRAIN 209-DOT RANDOM ESTIMATE 

A total of 45 segments was labeled for spring small grains; of these, 40 seg- 
ments were labeled for barley also. Proportion estimates at the segment level 
were obtained as simple random estimates from 209 dots of the 209-dot grid 
that was labeled. The results obtained for proportion errors at the state 
level are shown in table 5-1; the results at the segment level are shown in 
tables B-l(a) through B-l(d) in appendix B. These tables indicate that the 
small grains were underestimated slightly in all states except South Dakota. 
The overestimate in South Dakota is due to the overestimation of one segment 
which was overestimated by 15 percent as shown in table B-l(b). The barley 
estimate was within about 1 percent of the barley proportion obtained from 
ground truth. 


TABLE 5-1.- AVERAGE ABSOLUTE PROPORTION ERROR 


State 

Number of 
segments 

A 

P ■ 

■ P, percent 

S6 

S and B 

s 

B 

SG 

North Dakota 

22 

19 

-0.97 

-1.12 

-2.52 

South Dakota 

8 

7 

2.27 

-1.10 

0.81 

Minnesota 

8 

7 

-2.83 

0.51 

-2.16 

Montana 

7 

7 

-1.24 

-0.62 

-1.85 

All states 

45 

40 

-0.78 

-0.75 

-1.81 


Symbol definitions: 

SG - Spring small grains 
S - Spring wheat + oats + flax 
B - Barley 
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Highlights of the data shown previously in table 5-1 and tables B-l(a) through 

B-l(d) of appendix. B are outlined below. 

a. North Dakota 

(1) Estimates of spring small grains (22 segments) 

- The highest error occurred in agro-physical unit (APU) 21 where the 
individual segment estimates range from 2 percent to 15 percent 
under the ground-truth estimates. The average underestimate in 
this APU was about 9 percent, over nine segments. Most of this 
error was concentrated in three segments. 

- The remaining segments in North Dakota had proportion errors of 
less than about 6 percent with all but two having errors of less 
than about 3 percent. 

- The total cofnmission errors (138 dots) compensated for more than 
one-half of the omission errors (253 dots), and the resulting 
proportion error w^»s about -6.9 percent relative error when 
compared to the spring-grain ground truth. 

(2) Estimates of barley (19 out of 22 segments with a barley (B) window) 

- Proportion error in North Dakota barley estimates ranged from about 
-8 percent to +4 percent. Most of the barley dots that were 

mis identified were labeled S. 

- Some segments had a large relative error in the proportion of 
barley; typically, these were segments with small amounts of barley 
(0.5 percent to 7 percent). 

b. South Dakota 

(1) Estimates of spring small grains (8 segments) 

- Table B-l(b) in appendix B indicates that all but one segment had 
proportion errors which were less than 5 percent. The average 
error over the eight segments was +0.8 percent. However, 
segment 1154 was overestimated by 15.7 percent; it is an outlier. 

If this segment is removed from the averages, the remaining seven 
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segments result In an a/erage underestimation of 2 percent. This 
Indicates the same pattern of slight underestimation evident 1n the 
other three spring small-grain states. In segment 1154, most of 
the commission errors were due to winter wheat being called spring 
small grains. (See section 5.3 for causes of labeling error.) 

(2) Estimates of barley (7 segments) 

- The worst barley estimate occurred In segment 1784 where most of 
r.he barley was called other spring grains. This resulted In under- 
estimating the barley by 6.7 percent, which Is equivalent to having 
missed 78 percent of the barley In the segment. The remaining 
barley estimates were within 3 percent of the ground- truth estimate. 

Minnesota 

(1) Estimates of spring small grains (8 segments) 

- Spring small-grain estimates In Minnesota range from 1 percent to 
4 percent under the ground- truth estlmctei. The resulting under- 
estimation In acreage was about 2 percent. 

- Five of the eight sites had a relative error of less than 

10 percent. The main problem in the other three sites was spring 
wheat being called nonwheat. 

- The total commission errors (40 dots) compensated for more than 
one-half of the omission errors (74 dots), increasing the 
proportion estimate accuracy. 

(2) Estimates of barley (7 segments) 

- The barley estimates were, for the most part, very accurate (within 
1 percent with one exception), but there were only 20 barley dots 
to be found in all the segments in this state. Of these, only 11 
were identified correctly and the remaining nine were committed 
from mostly oats. 

- The barley was correctly estimated to be 0 percent in the two 
segments with no barley. 


d. Montana 


(1) Estimates of spring small grains (7 segments) 

- Montana estimation errors ranged from underestimation of about 
4 percent to overestimation of 0.5 percent. The average under- 
estimation was 1.8 percent. 

- Three segments out of the four with the worst underestimates 
accounted for most of the underestimation; all were from APU 22. 
These errors were caused mostly by mislabeling of sprint small 
grains as winter wheat due to early development. (See section 5.3.) 

(2) Estimates of barley (7 segments) 

- Barley estimate errors were low, ranging from a 3-percent under- 
estimate to a 1-percent overestimate. The relative error was 
quite high because of the low number of dots. There were only 
20 barley dots (about 10 percent of the spring small grains). 

Of the 20 dots, only 4 were correctly labeled. 

5.2 LABELING ACCURACY 

Labeling data for each spring small -grain category are tabulated in table C-2 
in appendix C; the results are presented at the state level for both the set 
of 45 segments that were labeled for spring small grains and the subset of 
40 segments that were labeled for barley. Tables C-2 (a) through C-2(e) con- 
tain segment level data. The information from these tables is summarized and 
depicted in the confusion matrices shown in table 5-2(a) through 5-2(e). 

These tables display the labeling accuracy and confusions for the spring small 
grains, tite barley, and the other spring-small grains. The accuracies for 
spring small grains are based on the set of 45 segments and the accuracies for 
barley and other spring small grains are based on the set of 40 segments. 

Tables 5-2(a) through 5-2(e) reflect that the overall labeling accuracy for 
those dots that were correctly identified as spring small grains was about 
90 percent for the four-state area and also for each state separately, with 
exception of South Dakota where it was 82 percent. The primary reason for the 
lower accuracy in South Dakota is that this state has a higher than average 
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TABLE 5-2.- SPRING GRAIN AND BARLEY LABELING ACCURACY FOR THE USN6P 
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TABLE 5-2,- Continued. 


fO 

•M 

O 

(0 

vO 





X 

W 

C 

+ 


ro 

O 

+ 


aj 

■s 



X 

>» 

cn 


ra 

0) 

c 


C 


u 




a. 


+ 

XI 

VI 


(/) 

'S 

T5 

<U 


ra 

p— 



o 

0) 

<i; 



X 

X 


+ 

rt3 











"3 

K 

X 


0) 

ro 

«3 


■i 


d 

>1 

Cn 

+ 

4- 

01 

c 





tn 

VI 

“u 

i. 

iji 


ro 

Q. 


JtJ 

X3 

VI 

o 

o 

■S 

•o 

<u 

+ 

+ 


T 

u 


<u 

0) 

ro 

ro 



(V 

OI 

IT3 

<o 

X 

45 

r— 


X 



>> 


cn 

a? 

0) 

C 

c 


f* 

4^ 

•p* 


1. 

u 

u 

ro 

« 

Q* 

CL 

CO 

03 


to 


III! 


CO 

1 • 


03 

</3 


r t 

* f 

CO 

GO 

to 

V) 



i*- 

<a 







c 

+ 

VI 





*A 


3: 

4.* 

c 

C 




• r» 

o 



0) 

£ 


4-> 


c 

2 

by 

»p* 


•p» 



e 


u 

S’ 

cn 

c 


pai 

Oi 

fF" 

♦r* 

Oi 

u 

c 

w 

u 

TO 

lO 

o 

o. 

o. 


CO 

s: 

00 

00 




1 

1 

O 

"1 

1 

1 

1 

1 

o 


03 

7Z 

00 

oo 




TABLE 5-2.- Continued. 



Barley B:B_ Barley labeled barley 

Hongrain + winter wheat + rye B:S — SarTey labeled spring wheat + oats + flax 

Spring wheat + oats + flax S:B — Spring wheat + oats + flax labeled barley 

Spring grains S:S — Spring wheat + oats + flax labeled spring wheat + oats + flax 



TABLE 5-2.- Continued. 



Barley B:B _ Barley labeled barley 

nongrafn ^ winter wheat + rye B:S _ Barley labeled spring wheat + oats + flax 

Spring wheat + oats + flax S:8 _ Spring wheat + oats + flax labeled barley 

Spring grains S:S _ Spring v/heat * oats + flax labeled spring wheat + oats *■ flax 



TABLE 5-2.- Concluded. 



Barley B:B — Barley labeled barley 

Nongrain + winter \vheat -*• rye B:S — Barley labeled spring wheat + oats + flax 

Spring wheat + oats + flax S:B — Spring wheat + oats + flax labeled barley 

Spring grains S:S — Spring wheat oats + flax labeled spring wheat + oats + flax 



proportion of barley with respect to the sprinci small grains coupled with a 
low barley labeling accuracy. However, the barley in this state represents 
only 10 percent of the barley in the four-state area; therefore, there was 
little effect on the overall labeling accuracy for the USNGP. 

Labeling of barley (given that the ground truth identifies a dot as barley) 
was not as successful as the overall labeling accuracy, and It turned out to 
be 56 percent for the four-state area. Note that the barley population is 
only 16 percent of the total spring small -grain population. Its distribution 
is such that it heavily overlaps the distribution of the other spring small 
grains. This overlap region represents a much higher proportion of the barley 
than that of the other spring small grains. Tlierefore, the labeling accuracy 
is correspondingly lower. 

On a state-by-state basis, the labeling accuracy for dots which were identi- 
fied as barley by the ground truth was about 60 percent for two states. North 
Dakota and Minnesota. Labeling accuracy for South Dakota and Montana was 
about 25 percent. These last two states contributed only 50 ground-truth 
barley dots from the 341 total barley dots. Tlius, North Dakota and Minnesota 
dominate the overall accuracy which ^ .*ns out to be 56 percent as mentioned 
previously. 

Most of the barley that was mislabeled was confused by the analysts as being 
other spring grains. This consisted of about 30 percent of the total ground- 
truth barley. The remaining 10 percent was called nonsmall grains. 

5.3 ERROR CHARACTERIZATIONS: SPRING SMALL GRAINS 

Error characterizations for this test were produced with procedures similar to 
those described in reference 9. Briefly, an analyst studied each labeling 
error and attempted to ascribe the error to one of a standard set of causes 
which had been identified previously. The standard set of causes for mis- 
labeling was determined from past experiences. The causes are explained in 
appendix D, table D-1. 
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A suimnary of the error characterization results for this test is given in 
tables 5-3(a) and 5-3(b). The error characterizations for each segment are 
found In appendix D, tables D-2(a) through 0-2(e). Some salient points that 
can be made from the results of the error characterizations for each state are 
outlined below. 

a. North Dakota 

In North Dakota, 60 percent of the omission errors were ascribed to an 
abnormal behavior of the crop as seen in the Landsat imagery. The abnor- 
mal behavior consisted of (1) late development of the crop which contri- 
buted 51 percent, (2) abnormal signatures (or unexpected colors) which 
contributed 7 percent, and (3) early development which contributed 
2 percent. In this state, more than one-half of the dots in omission 
errors ascribed to the late development of the crop came from three seg- 
ments: 1394, 1457, and 1920. The large majority of grain fields omitted 

in segment 1394 were extremely late; none were harvested on September 4; 
and only a few were harvested on September 21. In segment T457, the grain 
omitted was not entirely emerged on June 23 (a July 11 acquisition would 
have been highly beneficial), and the late fields looked like sumnier crops 
on August 16.‘ In segment 1920, the lateness of the grain in the southerly 
location and the adjusted and historical crop calendars for the area were 
misleading and were the major causes for the mislabeling. The historical 
Robertson biostage was 5.6 for the reference date; the Robertson biostage 
adjusted was 6.0; and the actual Robertson biostage was 4.4 according to 
18-day ground observations. The contribution to omission errors due to 
late grain in these three segments alone was about 4 percent. Thus, the 
labeling accuracy of spring small grains in North Dakota would have been 
89 percent instead of 85 percent if the late grain in these segments had 
been labeled correctly. 

The most noticeable commission error was that of nongrain following the 
temporal color sequence of small grain. An example of this error cause is 
the 42 dots or 2.5 percent committed to grain in North Dakota. Most of 
these dots (38) were committed to the category of other small grains 
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TABLE 5-3.- PERCENTILE SUMMARY OF ERROR CAUSES BY STATE 
(a) Omission errors 


Error causes 

North Dakota 

South Dakota 

Minnesota 

Montana 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Lack of acquisitions 

- 

- 

U 

24.4 

3 

4.1 

■5 

10.6 

Behind ACC, late developnent 

no 

51.4 

7 

15.6 

18 

24.3 

8 

17.0 

Ahead of ACC, early developnent 

5 

2.0 

- 


7 

9.5 

9 

19.1 

Abnormal small-grain signature 
(not expected color) 

17 

6.7 

1 

2.2 

16 

21.6 

6 

12.8 

Abnormal nonsinal 1 -grain 
signature (not expected color) 

- 

- 

- 

- 

- 

- 

- 


Subtotal 

152 

60.1 

19 

42.2 

44 

59.5 

28 

59.6 

Double cropping practice 
or weeds 

7 

2.8 

4 

8.9 

1 

1.4 

1 

2.1 

Volunteer wiieat 

- 

- 

- 

- 

- 

- 


- 

Detectable field destruction 

- 

- 

- 

- 

- 

- 

- 

- 

Nondetectable field destruction 

- 

- 

- 

- 

- 

- 

- 

- 

Subtotal 

7 

2.8 

4 

8.9 

1 

1.4 

1 

2.1 

Wrong acquisition used 
for label ing 


- 


” 

- 

- 


- 

Inadvertent error (clerical), 
inconsistency 

9 

3.6 

4 

8.9 

4 

5.4 

2 

4.3 

Subtotal 

9 

3.6 

4 

8.9 

4 

5.4 

2 

4.3 

Unlike other causesj 
segment unique causes 

17 

6.7 

4 

8.9 

5 

6.8 

4 

8.5 

Border or edge pixels 

39 

15.4 

4 

8.9 

14 

18.9 

5 

lO.Q 

Narrow fields 

28 

11.1 

8 

17.8 

5 

6.8 

2 

4.3 

Controversial ground truth 

1 

0.4 

2 

4.4 

1 

1,4 

5 

10.6 

Subtotal 

85 

33,6 

18 

40.n 

?5 

33.8 

11 

23.4 

Total omission errors 

253 

100 

45 

100 

74 

100 

47 

100 




TABLE 5-3.- Concluded, 
(b) Commission errors 


Error causes 

North Dakota 

South Dakota 

Minnesota 

Montana 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Lack of acquisitions 

1 

0.7 

- 

- 

8 

20.0 

- 

- 

Code 33 data deficiency 

6 

4.3 

- 

- 

3 

7.5 

- 

- 

Behind ACC, late development 

2 

1.4 

- 

- 

- 


2 

10.0 

Ahead of ACC, early development 

- 

- 

- 

- 

" 

- 

1 

5.0 

Abnormal small-grain signature 

- 

- 

- 

- 

•• 

- 

- 

- 

Abnormal nonsmall-grain 
signatures (Including winter 
wheat) 

42 

30.4 

41 

71.9 

9 

22.5 

5 

25.0 

Subtotal 

51 

37.0 

41 

71.9- 

20 

50.0 

8 

40.0 

Wrong acquisition used for 
1 abel 1 ng 

- 

- 

- 

- 

- 

- 

- 

- 

Inadvertent error (clerical), 
inconsistency 

30 

21.7 

3 

5.3 

- 

- 

2 

10.0 

Subtotal 

30 

21.7 

3 

5.3 

- 

- 

2 

10.0 

Some differences could be 
detected between this signature 
(color sequence) and other 
spring small-grain signatures; 
unlike other causes; segment 
unique causes (some look like 
spring small grains, but some 
do not) 

23 

16-7 

3 

5.3 

16 

40.0 

5 

25.0 

t 

Border or edge pixels 

23 

20.3 

7 

12.3 

3 

7.5 

1 

5.0 

Narrow fields 

5 

4.3 

2 

3.5 

1 

2.5 

2 

10.0 

Controversial ground truth 

- 

- 

1 

1.8 

- 

- 

2 

lO.Q 

Subtotal 

57 

41.3 

13 

22.8 

20 

50.0 

10 

50.0 

Total commission errors 

138 

100 

57 

100 

40 

100 

20 

100 
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(spring wheat + oats + flax). The highest errors were caused by sunflower 
fields (10 dots) and canary seed fields (9 dots) that looked like spring 
wheat. The sunflowers committed from one segment represented 55.6 percent 
of all sunflower dots comnitted to spring grains in North Dakota and 
50 percent of all sunflower dots committed to spring grains in the four- 
state area’. Thus, most of the commission error problems of committing 
sunflowers to wheat came from one segment. There was only one site in 
which canary seed was labeled spring small grains. 

b. South Dakota 

Reasons for the omission errors in South Dakota were fairly well distri- 

* 4 

buted among the list of error causes. The largest single source of error 
was ascribed to lack of a key acquisition that would have enabled the 
analyst to identify some of the spring small grains. Specifically, dots 
in this error category did not show detectable emergence in the available 
acquisitions in any of the three segments in which this type of error 
occurred. Thus, spring smalNgrain labels would have had to be ascribed 
to some dots on the basis of its preemergence and the all -ripe or harvest 
acquisitions. 

The commission errors in South Dakota were due to causes discussed in 
section 5.1. .The error committed in segment 1154 was that of labeling 
winter-wheat dots as spring small grains. Thirty-three out of the 41 dots 
in this error category occurred in this segment. The reasons for this 
error follow. First, a black-looking signature in the early spring was 
interpreted to be abandoned and replowed winter wheat. Second, the 
historical crop calendar indicated that winter wheat should have been 
harvested at a time when, in reality, it was still green vegetation. 

c. Minnesota 

The major omission error causes in Minnesota were late development of the 
spring small grains and abnormal signatures. The omission error from 
these causes was 45 percent of the total. 
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The major cause for commission errors in Minnesota was confusion caused by 
the similarity of nongrain signatures to spring small-grain signatures. 
Causes for the similarity of signatures in many cases could not be deter- 
mined. Some signatures, such as corn signatures In segment 1524, were 
exactly the same as small-grain signatures in all acquisitions (abnormal 
nonsmall-grain signature). Some signatures, such as corn In segment 1566, 
were similar in all but one or two of the acquisitions (abnormal signa- 
tures, segment unique). In this latter case, an Incorrect assumption 
would be made*, for example, an expected plowed signature was not observed 
because the crop was plowed a little earlier or later than expected. 

d. Montana 

Reasons for omission errors In Montana, as in South Dakota, were fairly 
evenly distributed among the list of error causes. In Montana, however, 
the total omission error was 23 percent, whereas in South Dakota, it was 
28 percent. The highest source of error was ascribed to the combination 
of early and late development of the small-grain crops. Late crops were 
confused with summer crops; early crops were confused with winter wheat. 
There was also some confusion with idle fallow due to the fact that some 
small-grain fields never reached the color which is typical of green 
vegetation. 

Commission errors in Montana were, for the most part, distributed evenly 
among the causes, although the two largest contributors to commission 
errors were caused by signatures that v/ere similar to those of small 
grains. 

!• 

5.4 ERROR CHARACTERIZATIONS; BARLEY 

In order to determine the degree of success of the procedure to separate 
barley from the other spring small grains, a technique was used which 1s simi- 
lar to that used for characterizing errors in labeling spring small grains. 
Specifically, an attempt wcTs made to ascribe each error to a standard set of 
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previously defined causes, A tabulation of the errors and error causes can be 
found in cable 5-4 along with information regarding the acquisition and 
decision boundary selection. 

.The barley procedure contains two steps which are key to the success of the 
separation process; the selection (by -an analyst) of the acquisition in which 
barley can be separated from the other small grains and the placement on a 
green number versus brightness scatter plot of a decision boundary to reflect 
the separation as perceived by the analyst. In order to assess how well these 
two steps were executed by the analyst, the evaluators used ground truth to 
label scatter plots of the separation acquisition that was actually used by 
the analyst plus ground truth of the previous and the following acquisitions. 
The evaluators drew a decision boundary that would best separate barley from 
the other small grains on the ground-truth plots. Those dots that had differ- 
ent labels than those determined by the original analyst were then recorded. 
The results of this exercise are also shown in table 5-4 for each of the other 
small -grain crops which were originally mislabeled as barley and those for 

barley originally mislabeled as other small grains. 

» 

Table 5-4 indicates that only about 10 percent of the other small -grain pic- 
ture elements (pixels) that were mislabeled as barley can be attributed to the 
selection of the decision boundary and none to the selection of the separation 
acquisition. Conversely, 2 percent of the barley pixels mislabeled as other 
small grains are attributable to the boundary selection and only 1 percent to 
the acquisition selection. Therefore, neither the selection of a decision 
boundary nor the selection of a separation acquisition appears to have been a 
major cause of error in the separation of barley from the other small grains. 

Table 5-4 also shows that the major cause for mislabeling other small grains 

« 

as barley was the typical barley-like behavior of these pixels. That is, they 
acquired a brighter, less red signature in the separation acquisition; or, 
alternately, they migrated toward the lower right of the scatter plot. In 
some cases, it was determined that this happened because of hail damage; but 
in most cases, no apparent reason could be found. Thirty- four pixels or 
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TABLE 5-4.- ERROR CHARACTERIZATIONS FOR LABELING OF BARLEY 
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60 percent of the labeling errors of other spring small grains as barley were 
due to this one problem. The second most prevalent cause for mislabeling 
other spring smaP grains as barley was t'le early development of some of the 
spring grains which caused the signatures to display a turning appearance at 
the same time as "normal'' barley. Ten pixels or 18 percent of the errors fall 
into this category. It should be noted that the difference between this error 
category and the one described above 1s that the ground truth did confirm the 
earl 1 ness of the development of the crop In the latter case but did not In the 
earlier case. 

The main cause for labeling errors in which barley was labeled as other spring 
small grains was the late development of barley. Fifty percent of the errors 
or 53 out of the 110 mislabeled dots were due to this problem. Thus, 15 per- 
cent of the barley, which these 53 dots represent, 1s essentially the propor- 
tion of the ground-truth barley for which the assumption that barley matures 
earlier than wheat was not confirmed by the test. 

The second most prjdominant cause for labeling barley as other spring small 
grains was that barley acquired a very weak turning signature at the expected 
time. Thus, the analysts could detect the turning signature (little brighter, 
less red, more pink) with the aid of ground truth but could not detect It at 
the time in which the, original analysis was done. A similar statement can be 
made with respect to the scatter plot. That Is, the dots had only partially 
migrated to the barley region of the plot but were mostly mixed with spring- 
wheat dots. In this case, the ground-truth annotation did not confirm that 
these barley dots were late in development, although many of them may have 
been. 


Finally, It should be noted that flax was never confused with barley, whereas 
oats was the spring small-grain crop most likely to be labeled barley. 

5.5 ERRORS BY CROP CATEGORY 

Computations in table 5-5 were done to determine If any one small-grain crop 
is more likely to be omitted than others or If any one crop is more likely to 


TABLE 5-5.- CAUSES OF OMISSION AT THE SPRING SMALL-GRAIN LEVEL BY GRAIN 

CATEGORY FOR THE USNGP 



Number of 
dots In 
category 

Labeling error cause 



Crain category 

a 


^2 

e 

p 

®1 

‘2 

n 

k 


0 

Total 

Percent 

omitted 

Spring wheat and 
durini wheat 

1602 , 

9 

98 

12 

17 

6 

15 

10 

38 

12 

15 

6 

238 

14.86 

Oats 

356 

7 

32 

8 

9 

6 

11 

3 

10 

5 

8 


99 

27.81 

Flax 

57 

1 

20 


10 


2 


2 




35 

61.40 

Barley 

360 

2 

13 

1 

4 

1 

2 


12 

2 

7 

3 

47 

13.06 

Total spring small 
grains 

2375 

19 

163 

21 

40 

13 

30 

13 

62 

19 

30 

9 

419 


Contribution to the 
total error for each 
error cause (proportion 
of all pixels In the 
ground-truth category) 

• 

0.8 

6,86 

0.88 

1.68 

0.55 

1.26 

0.55 

2,61 

0.8 

1.26 

0.38 



17.64 


Symbol definitions; 

a A lack of Informative acquisitions (useful to labeling) contributed to the labeling error. 

Yj Crop was behind the ACC, late planting and development. 

Yj Crop was ahead of the ACC, early planting and development. 

The dot was on a single narrow field; the signature may or may not have been overridden by surrounding 
signatures of other categories. 

eg The dot encotnpassed multiple strips so narrow that they presented a single, Integrated signature. Field width 
was not resolvable by the Landsat scanner. 

0 The signature of the target crop did not follow the expected temporal color sequence of small grain throughout 
the acquisition. 

\ Clerical error — the error-pixel clearly followed a temporal sequence for Its category. Since other dots with 
the same temporal sequence were consistently identified correctly, then this error was a clerical error. 

11 Double cropping practice of a second crop or weeds bec»ne the dominant signature and caused an Increase in the 
infrared response after grain harvest. 

It Border or edge pixel — spectral and spatial confusion occurs between two or more fields of different types due 
to misregistration of acquisitions or confusion as to field size or shape. 

<3 The author disagrees with the ground-truth overlay label, but It was reluctantly accepted as being valid. 

♦ Segment-unique causes -the error did not match any of the causes of a general nature. The pixel had 

insufficient spectral Infonnotlon to cause correct categorization, but some correlation could be made with the 
correct category after the ground truth was known. 
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contribute to a particular error cause than others. The error causes are 
broken down for each of the crops in this table. 

Data from table 5-5 indicate that flax is the crop most likely to be omitted 
and with a 61 percent probability. The main cause of mislabeling is the late- 
ness of the crop and consequent confusion with summer crops. About two- thirds 
of the omissions were due to .this cause. The other one-third was due to 
abnormal signatures, namely, signatures that never attained a good green 
vegetation signature (red, light red, and orange). 

The second most likely small-grain crop to be omitted is oats. This omission 
occurred for about one-third of the pixels or 28 percent. The main cause of 
mislabeling was also the lateness of the crop. 

Because of its abundance, the largest contributor to the omission errors was 
spring wheat, lateness being the main cause. Sixty-two percent of the small - 
grain pixels omitted were spring wheat. Note, however, that the percentages 
of wheat omitted (15 percent) and barley omitted (13 percent) were much lower 
than those for the other categories. 

Late development of the spring small -grain crops and their consequent con- 
fusion with summer crops was the largest single cause of omission. This is 
true for the spring small grains as a whole and also for each individual crop. 
It contributed 7 percent out of the 18 percent total omission errors. 

Table 5-6 is a summary of the relative importance of the lateness of a crop as 
a cause of omission errors for each of the individual spring small-grain crops 

5.6 SEPARATION OF BARLEY BY GEOGRAPHIC REGIO N 

The separability of the barley from the other spring small grains was deter- 
mined from ground truth by placing an optimum decision boundary between these 
two categories in the scatter plot of the acquisition with the most separation 
The separ'ability was then defined to be the percent of barley pixels on the 
barley side of the decision boundary. 
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TABLE 5-6.- RELATIVE IMPORTANCE OF LATENESS OF THE 
CROP AS CONTRIBUTOR TO THE OMISSION ERROR 


Grain 

Percent of the crop 
omitted due to 
lateness of the crop 

Percent of the crop 
omitted due to 
all causes of error 

Flax 

Oats 

Wheat 

Barley 

35 (20 of 57 dots) 

9 (32 of 356 dots) 

6 (98 of 1602 dots) 

4 (13 of 360 dots) 

61.4 (35 of 57 dots) 

27.8 (99 of 356 dots) 

14.9 (238 of 1602 dots) 
13.1 (47 of 360 dots) 


Figure 5-1 is a map of the four-state area depicting the locations and 
separability of segments that were processed. Segments coded with a red 
circle had barley separability of less than 60 percent. Thus, it can be seen 
that in western Minnesota and southeastern North Dakota is an area in which 
there is a preponderance of segments with barley separability greater than 60 
percent. Otherwise, there seems to be no other well-defined geographic 
patterns. It should be noted, however, that other areas of high accuracy may 
exist, but the map would not show them because of the lack of sufficient sites 
to show a pattern. For example, figure 5-2, which depicts the separability of 
1977 crop year data, appears to indicate that there is a region in Montana 
where the separability is also greater than 60 percent for six out of '=even 
sites. In 1978, there was only one site in the same region. Thus, at best, 
there are indications that barley can be identified with high accuracy for 
segments along part of the North Dakota and Minnesota border. 

Another observation that should be noted is that the preponderance of seg- 
ments, which were used for the development of the barley separation procedure, 
also occur in the area of western Minnesota and southeastern North Dakota 
(fig. 5-2). Since the data used for development were from the 1977 crop year, 
this fact could be taken as an example of extendabil ity from one year to 
another, at least for that area in which there are indications that barley can 
be identified. Areas shown to have separability in 1977 are indicated in 
figure 5-1 with brown outlining for comparison. 
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Finally, figure 5-3 is a map that depicts the relative concentration of 
barley-growing regions in the four-state area. It can be observed that the 
area of western Minnesota and eastern North Dakota are also areas where high 
concentrations of barley exist. 
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Figure 5-3.- The barley producing areas of the United States 



6. CONCLUSIONS 


The main objective of this investigation was to assess the success of the TY 
procedure to separate and label barley and the other small grains. Results of 
testing the procedure indicate that there was no particular difficulty in 
exercising. the procedure correctly whenever it was applied. Decision lines 
were drawn on the plot of the appropriate acquisition using ground-truth 
labels, and the decision boundary was found to be very little different from 
the one provided by the analysts supporting this study. 

Because the labeling accuracy for spring small grains obtained by the evalua- 
tors turned out to be somewhat different from that obtained by the TY 
analysts, some comments are in order. Specifically, the average accuracy for 
labeling spring small grains in the four-state area was about 70 percent in TY 
(ref. 10) and about 82 percent when obtained as part of the work conducted in 
this investigation. It is apparent that the difference in accuracy is domin- 
ated by the difference in the labeling for North Dakota. Overall labeling 
accuracies in the other states were comparable with those in TY with one 
exception in South Dakota. The accuracy in that state was affected by a large 
commission error that occurred in one segment in which late winter wheat was 
labeled spring wheat. It was outside the scope of the present investigation 
to search for the reason, for the difference in accuracy. However, the follow- 
ing argument can be made; the regionalization of the segments to be labeled 
as a different approach to the analysis, the inclusion of trend analysis of 
past cropping practices as part of the labeling process (ref. 11), and the 
removal of any time constraints for segment processing influenced somewhat the 
final outcome of the labeling accuracy obtained. 

Another subject to be addressed regarding the labeling of small grains 
includes the problems identified that were found to affect the accuracy of 
labeling. The major problem consists in the confusion that results when crops 
develop at different times or at different rates from that expected by the 
analysts for ''normal'' crops. Thus, late winter-wheat fields were confused 
with spring grains, and late spring grains were confused with summer crops. 
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It was not determined if the error was caused by normal variations from the 
average crop calendars or if the crop calendars themselves were in error. In 
either case, the deviations were usually of the magnitude of 2 weeks and as 
high as 4 weeks. Also in the former case, the analyst did not have, for the 
most part, the information that would allow one to assess the. variability of 
the crop's development stages around the average. Some state level 
information (from state agricultural reports) regarding crop stages was used. 

A variation of these types of problems occurred when nonsmal 1 -grain crops fol- 
lowed approximately the same temporal pattern as the small-grain crops. This 
crop pattern was apparently the case for some of the commission errors, 
especially between sunflowers and spring grains. In this case, analy-ts 
attempted to detect differences in signatures but were not always successful. 

From an overall point of view, the barley separation procedure turned out to 
be successful for the labeling of spring grains. Namely, given a pixel was 
correctly labeled as spring grains, the probability was around 90 percent that 
it would be correctly labeled either barley or other spring- small grains. 

The procedure was only partially successful at detecting and labeling bar- 
ley. The probability of correctly labeling barley pixels was only 56 per- 
cent. The barley labeling problems were caused, at least in part, by the 
fact that barley did not always exhibit the expected behavior, particularly in 
South Dakota and Montana. Another problem with the accuracy of barley label- 
ing was simply the extensive overlap of the spactral distribution of barley 
with the distribution of the other small grains. This problem coupled with 
the fact that the population of barley is small compared to the population of 
the other small grains results in the proportion of barley pixels in the over- 
lap region being very high with respect to the barley population. This is 
not the case with the other small grains. The successful ness of the proce- 
dure can be summarized by concluding that the assumptions upon which the pro- 
cedure is based are not necessarily true; barley does not always mature prior 
to other spring small grains and the assumption that barley becomes brighter 
and less green within a definite growth stage range is also not necessarily 
true. 



There is a region in western Minnesota and Southeastern North Dakota, however, 
where the barley procedure appears to work well. In this region, the separa- 
bility (as defined in section 5.6) was 81 percent. Also, a high density of 
barley is grown there. Thus, it appears that the extendability of the proced- 
ure is confined to this region of high density barley. The extendability to 
the remaining parts of the four-sta.te area is tenuous at best. 

Further development of the barley procedure should include further investiga- 
tions into the similarities and differences in planting and development of 
barley and the other small grains and the effect of geographic, functional, 
and meteorological influences. Also, the development of methodologies for 
detecting these differences when they exist should also be included. 
Specifically, it appears that when barley is planted late with respect to the 
other small grains, it becomes more difficult to detect with the current pro- 
cedure but may still be detectable. Conversely, early developing (or early 
harvested) wheat or oats will normally be called barley. 
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APPENDIX B 


PROPORTION ESTIMATES FOR ALL THE SEGMENTS IN THE STUDY 



TABLE B-1.- SPRING SMALL-GRAINS 209-DOT RANDOM ESTIMATES 



'firoufKl-truth proportions computed fraii the 209-dot groimd-tnith labels (section 2.2). Results presented in 
this table :iiay differ frort any previously presented evaluation results based on digitized ground truth. 
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Ground-truth proportions computed from the 209-dot ground-truth labels (section 2.2). Results presented in 
this table may differ from any previously presented evaluation results based on digitized ground truth. 
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Ground-truth proportions computed from the 209-dot ground-truth labels (section 2.2). Results presented in 
this table may differ from any previously presented evaluation results based on digitized ground truth. 



TABLE B-1.- Concluded. 
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APPENDIX C 


DOT LABEL DATA FOR ALL SEGMENTS 
IN THE STUDY 


TABLE C-1.- SYMBOL DEEINJTIONS FOR TABLES C«2(a) THROUGH C-2(e) 


Symbol 

Definition 

B;B 

Barley labeled barley 

B:N . 

Barley labeled nongrain + winter wheat + rye 

B:S 

Barley labeled spring wheat + oats + flax 

F:B 

Flax labeled barley 

F:N 

Flax labeled nongrain + winter wheat + rye 

F:S 

Flax labeled spring wheat + oats + flax 

N:B 

Nongrains + winter wheat ■<' rye labeled barley 

N:N 

Nongrains + winter wheat + rye labeled nongrains + winter wheat + rye 

N:S 

Nongrains + winter wheat + rye labeled spring wheat + oats + flax 

N:SG 

Nongrains + winter wheat + rye labeled spring grains 

0:B 

Oats labeled barley 

0:N 

Oats labeled nongrain + winter wheat + rye 

0:S 

Oats labeled spring wheat + oats + flax 

0:SG 

Oats labeled spring grains 

SG:N 

Spring grains labeled nongrain + winter wheat + rye 

SG:S6 

Spring grains labeled spring grains 

SW:B 

Spring wheat labeled barley 

SW:N 

Spring wheat labeled nongrain + winter wheat + rye 

SW:S 

Spring wheat labeled spring wheat + oats + flax 

SW:SG 

Spring wheat labeled spring grains 




TABLE C-2.- SPRING SMALL-GRAIN DOT-LABEL DATA FOR THE USNGP 

$a) The four-state area 



The lumber in the denominator represents the total of ground-truth dots in the category being labeled, 



































TABLE C-2,- Continued 
(b) North Dakota 
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TABLE C-2,- Continued, 
(c) South Dakota 
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The nunber in the denominator represents the total of ground-truth dots in the category being labeled. 




‘The nunber ia the denominator represents the total of ground-truth dots in the category being labeled 


























































































TABLE C-2.- Concluded 
(e) Montana 




































































APPENDIX D 


CHARACTERIZATION OF LABELING ERRORS 



TABLE D-1.- SYMBOL DEFINITIONS FOR TABLES D-2(a) THROUGH D-2(e) 


Symbol 

Definition 

a 

A lack of Informative acquisitions (useful to labeling) contributed 
to the labeling error. 

Code 33 

A nongrain crop was committed to barley because of a holiday fn 
Landsat coverage of the field from planting to senesence/harvest. 
A detectable infrared response was not present In any of the 
available acquisitions. 


Crop was behind the ACC, late planting and development. 

Y2 

Crop was ahead of the ACC, early planting and development. 

n 

The dot was on a single narrow field; the signature may or may not 
have been overridden by surrounding signatures of other categories. 

"2 

The dot encompassed multiple strips so narrow that they presented a 
single, integrated signature. Field width was not resolvable by the 
Landsat scanner. 

6 

The signature of the target crop did not follow the expected temporal 
color sequence of small grain throughout the acquisitions. 

X 

Clerical error— the error-pixel clearly followed a tanporal sequence 
for its category. Since other dots with the same temporal sequence 
were consistently identified correctly, then this error was a 
clerical error. 

u 

Double cropping practice of a second crop or weeds became the 
dominant signature and caused an increase in the infrared response 
after grain harvest. 

V 

The signature of the nonsmall grain followed the expected temporal 
color sequence of small grain throughout the acquisitions. 

Tf 

Border or edge pixel — spectral and spatial confusion occurs between 
two or more fields of different types due to misregistration of 
acquisitions or confusion as to field size or shape. 

0 

Author disagrees with the ground-truth overlay label, but it was 
reluctantly accepted as being valid. 


Segment-unique causes — the error did not match any of the causes of 
a general nature. The pixel had insufficient spectral information to 
cause correct categorization, but some correlation could be made with 
the correct category after the ground truth was known. 

“1 

Field destroyed by grazing, plowing, or discing. Analyst should be 
able to detect destruction of the field. 

“2 

Field destroyed by grazing, plowing, or discing. Analyst should not 
be able to detect destruction of the field. 

0 

Omission error 

C 

Commission error 


D-1 




































































*Refer to table D-1 for definitions of syabols 





















































TABLE D-2.- Continued 

































































































































































